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In any Markov chain with finite state space the distribution of transition records always belongs to 
the exponential family. This observation is used to prove a fluctuation theorem, and to show that the 
dynamical entropy of a stationary Markov chain is linear in the number of steps. Three applications 
are discussed. A known result about entropy production is reproduced. A thermodynamic relation 
is derived for equilibrium systems with Metropolis dynamics. Finally, a link is made with recent 
qq ■ results concerning a one-dimensional polymer model. 
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. Introduction The fluctuation theorem, discovered in 1993 by Evans, Cohen and Morriss pJjWas the start of a 
series of new developments in the study of stationary non-equilibrium systemsfsee for example [2| for an overview). 
A formulation of the fluctuation theorem in stochastic context was given in [1, 0| • These authors showed that the 
theorem holds in full generality for any stationary probability distribution of any Markov process. The proof given 
here is restricted to Markov chains with finite state space. In this way technicalities can be avoided. 

The method of proof of the present paper is based on the observation that a certain probability distribution, 
hereafter called distribution of transition records, always belongs to the exponential family. From this result the 
fluctuation theorem follows immediately. Indeed, Lebowitz and Spohn Q and Maes [H already observed that a Gibbs 
' distributed random variable is all what is needed to derive a fluctuation theorem of the Gallavotti-Cohen type 0] in 
, a stochastic context. This point has been further elaborated in Q. 

The motivation for studying the distribution of transition records was twofold. At one hand the present authors 
had noticed @, S| that, in the Markov chain of increments of some model of random walk describing a polymer, this 
distribution belongs automatically to the exponential family. At the other hand, it is a straightforward generalization 
of a distribution introduced in recent work of Carati [1, [Tq] . This author assumed a Poisson distribution based on 
the assumption that subsequent visits to macrocells in phase space of a system of classical mechanics are mutually 
independent. 

Definitions Throughout the paper we consider a fixed state space T, containing a finite number N of states. A 
Markov chain with state space T is determined by initial probabilities p(x), with x in T, and by transition probabilities 
"■^J w(x,y), with x and y in T. The probabilities p(x) are used as initial values for the equation of motion 
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Pt+i{x) = ^2pt(y)w(y,x). (1) 
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t^J" , The probability distribution p(x) is stationary if 
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p( x ) = ^2p(y) w (y, x ) ( 2 ) 



in 

oo 

o 

O p(x)w(x,y) =p{y)w(y,x) (3) 
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for all x G r. A stronger condition is detailed balance 



for any pair of states x, y 6 T. Summing this equality over y yields 

The w(x,y) are parameters of the Markovian model. However, they are not independent because of the normaliza- 
tion condition ^2 y w(x, y) = 1. Introduce therefore independent parameters 9 XtV and dependent r\ x such that 



w(x,y) = wo(x,y)e 0x - y if x ^ y 

w(x,x) = e~ n *, (4) 

with enabling factors Wo(x,y) satisfying wo(x,y) = Wo(y, x), and Wo(x, y) = 1 or wo(x,y) = 0. The parameters r\ x 
are considered to be functions of the & x ,y For convenience, let 9 X , X — and wq(x,x) = 1 for all x. 
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A Gibbs distribution for transition records The record of transitions A: is a sequence of numbers one for each 
pair of states x, y, counting how many times the transition from x to y is contained in a given path of the Markov 
chain. Our key quantity is the probability d x (k) that a path starting in x results in a given record of transitions k. 

It is not difficult to see that one can write 

d x (k) = c n (x,k) Y[w(y 7 z) k y-' 

= c n (x,k)exp(-^ e {k)). (5) 

In this expression, c n {x, k) counts the number of paths that have the same record of transitions k and that are allowed 
by the enabling factors wo(x,y). The dynamical entropy variable ^${k) is given by 

^e(k) = ^2vyky, y +^20y,zk y , z . (6) 

y v,x 

In particular, ([5]) implies that the distribution d x (k) belongs to the curved exponential family. 
The special form of {SJ makes it easy to calculate the following identities 

- _( k ) i w ^ v h h \ (7) 

W(U,U) 



with 



This can be written as 



(k u ,v)x — ^ ] d x (k)k UtV . (8) 



w{u 1 u)(k u ^ v ) x = (k u>u ) x w(u, v). (9) 

The latter result is clearly also valid for u = v. This means that (k UtV ) x , the average number of transitions from u to 
v, is proportional to w(u, v), the probability to go from u to v. 

Linear production of dynamical entropy A path 7 = (xq, Xt, ■ ■ ■ x n ) of the Markov chain has probability p(xo)w(pf) 
with 



7(7) = w(x ,xi) ■ ■ ■ w(x n -i,x n ). (10) 

i denote the initial and 7/ = a 

defined by 



Let 7^ = xq denote the initial and 7/ = x n the final state of 7. The dynamical entropy So of the Markov chain is 



Sj n) = -5>(7iM7)h«>(7). (11) 
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The time-reversed dynamical entropy fllj ] is defined by 

7 

= -^2p{lf)w(l)^w{j), (12) 

7 

where 7 is the reversed path 7 = (x n , x n -\, ■ ■ ■ xq). 

Observe that the dynamical entropy So can be expressed in terms of the distributions d x (k). Indeed one has 



S e = -J2p( x )J2( k y^)x^nw(y,z) 



y,z 



(13) 



If p{x) is stationary then this expression can be further simplified. One expects intuitively that each occurrence of 
some state y contributes to the dynamical entropy production with an amount I y , defined by 

I y = -^2w(y,u)\nw(y,u). (14) 

u 

Indeed, this is the result of Theorem Q] below. 

The corresponding relations for the time-reversed paths are 

X 

with Wg(k) = ^e{k), k x . y = k y<x , and 

I y = -^2w(y,u)lnw(u, y). (16) 

u 

Given a path with record of transitions k, the number of occurrences of the state y (neglecting the final state) is 
J2 z ky,z- The probability distribution of the latter quantity is the distribution studied by Carati [1, OH. Using 
the average number of occurrences of state y is 

^2(ky, z )x = w ^ y ^ ( k v,y)x- ( 17 ) 

The average entropy produced when leaving y is I y . Hence 

is the average production of dynamical entropy per step. One therefore expects the following result. 
Lemma 1 Assume w(x,x) > for all x. Then one has 

= E^^vV^' (is) 
= E^^vW^. ( 2 °) 

Next, one proves by full induction that 
Lemma 2 Let p be a stationary probability distribution. Then 

^2p(x){ky t y) x = nw(y, y)p{y). (21) 

X 

Combining the different pieces gives the result 
Theorem 1 Assume p is stationary. Then 

X 

S(? = n^2p(x)l x . (23) 

X 

Note that the condition of Lemma 1 that w(x,x) > for all x is not essential for the theorem to hold. It can be 
removed by a limiting procedure. 



Fluctuation theorem Without any assumption about the Markov chain, the probability distribution d x (k) belongs 
automatically to the exponential family. As noted in the introduction, this observation suffices to derive a fluctuation 
theorem. 

Let C(x, k) denote the class of all paths that start in state x and have the same record of transitions k. The 
probability of this class is d x (k). It is easy to see that all paths of this class have the same final state, denoted f(x, k). 
This implies that there is a one-to one-correspondence between paths of C(x, k) and reversed paths belonging to 
C(f(x, k), k). Introduce the entropy production variable 

w{l) = ]n ppMA (24) 

(called action functional in 0]). Note that, because of (0), W("f) is constant on the class C(x, k), with value 

In£M_tf fl (fc) + tf s (fc). (25) 

It is now straightforward to derive the fluctuation theorem 

Theorem 2 In any Markov chain with finite state space the entropy production variable satisfies 

Prob(VK = K) K 



Yvob{W = -K) 



(26) 



Proof 

One has 



Yiob{W = K) = ^p{li)w{~f)8 {w{l)}K} 

7 
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= e K ^2p(j l )w(j)6 {w{ - hK 



} 

= e. K ^p{-ii)w{-i)5 {w{ , t) _ K} 

7 

= e K Prob(W = -K). (27) 

□ 



In a more general setting [!, Hj], the same result has only been proved to hold asymptotically for large times. 
Entropy production The standard and the time-reversed dynamical entropies are related by 

S ( e n) ~ Sj n) = (* e ) - 

= (W0-J>(7>(7)ln^. (28) 

Application of the fluctuation theorem then gives 



2 ^ P(lf)w(j) 
= 4 n) -4 n) +E^W7)lrx P(7i) 
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Pilf) 



= KPvob(W = K) 

K 

= i^Prob(VK = K)K{1 - e~ K ) 

K 

> 0. (29) 



Of course, this positivity can be directly deduced from the first line of the above expression. 
On the other hand, using Theorem [T] and assuming stationarity, it is possible to write 



X 

with 

AS = ^^2(p(x)w(x,y) -p(y)w(y,x)) 
x,y 

idn ^ri (31) 
p{y)w(y,x) 

AS is the entropy production 0, HH, H2] . It vanishes for distributions satisfying the detailed balance condition. 

A one-parameter model of equilibrium states We now restrict ourselves to equilibrium, which means that proba- 
bility distributions satisfy the detailed balance condition. We start from a symmetric matrix a XjV labelled by x, y in 
the state space T. It will be used below to fix the transition probabilities w(x,y) of the model. We fix a parameter 
£, which is the analogue of the inverse temperature (3. In fact, in the case of Metropolis dynamics the matrix a x . y is 
given by 

a x , y = max{iJ(x),-ff(y)}, x^y 

^x \ 

.-0B{ X ) _Y^ WQ { x ,y)e-^v\ (32) 
y / 

where H(x) is the Hamiltonian, £ equals j3, and c = x wq(x, y) independent of x. 

Introduce a partition function S(£), a probability distribution p(x), and transition probabilities w(x,y), given by 

2(0 = 5>o(x,2/)e-« a -, (33) 

"(*,v) = ^A^- (35) 

Clearly, a x . y — a VtX implies that the detailed balance condition is satisfied. It is now straightforward to calculate the 
distributions d x (k). They are given by ([5]), with 




^e(k) = £ ^ k x , v a x ,y +^ I ^ K,y \np(a 

x,y x \ y J 



+n'La(0. (36) 
A short calculation, using |3^|) and the results of the section on linear production of entropy, gives 

]nZ(0=S(p) + ^ n) -Z(a). (37) 

Here, S(p) is the Boltzmann-Gibbs entropy, and (a) is the path average of the matrix a x>y . Expression (|37[) should 
be compared with the well-known result for the Massieu function lnS(/3) = S — f3U, where U = (H) is the average 
energy. This comparison shows that in (|37|) the thermodynamic entropy is replaced by the sum of two contributions, 

a static S(p) and a dynamical —Sg n \ Thermodynamic relations similar to (f3"T)l are derived by Carati [1, GI|. Here, 
we do not need a Legendre transformation to define entropy. In addition, (|36[) contains a path dependent term, which 
is absent in 0, [l(| ■ 

Related results are found in [l2l Il3l|. where the parameter s, appearing in the dynamical partition function, is the 
argument of a Laplace transform. Its role is comparable to that of our parameter £, which controls the transition 
probabilities of the Markov chain. 



A model of random walk Let us finally make the connection with the non-Markovian random walk model of [7|, [8j . 
The position after n steps is denoted X n . The increments x n = X n+ \ — X n are Markovian with transition matrix 

-(;./-;)■ <»> 

The two states are denoted + and — . The enabling factors wo(x,y) are all equal to 1. 

Let Xq = 0. The position X n after n steps is related to the record of transitions k by X n = k ++ + k |_ — k — k^ 

(number of steps to the right minus number of steps to the left). The energy of the polymer may be assumed to 

be proportional to k^ + k which is the number of changes of direction. Finally, let A = k^ — k [_ and note 

that A is either or ±1. Together with the identity n = k ++ + k |_ + k + k^ , this means that the record of 

transitions k can be expressed in terms of the physical quantities energy and position of the end point, up to an 
error A. This observation was used in @, [1] to prove that the joint probability distribution of energy and position of 
endpoint automatically belongs to the exponential family, be it with 3 parameters instead of 2. As a consequence of 
the latter, the distribution is only approximately that of Boltzmann-Gibbs, with a small error which is negligible in 
the limit of large n. 

Discussion We have shown that some known results about Markov chains can be formulated in terms of the 
distributions d x (k) which give the probability of a transition record k for paths of the Markov chain starting in state 
x. These results are connected with what is known as the fluctuation theorem. In addition we have pointed out, 
without going into much detail, that recent work of Carati [j| |T(|, and of the present authors @, Q involves the 
same or related probability distributions. We expect many more applications of our approach. We believe that the 
distribution of transition records d x (k) will be the preferred tool, rather than the fluctuation theorem (Theorem [2]). 

Like in [Til Il5j , the fluctuation theorem and the extensivity of the entropy production, as proved here, hold for 
arbitrary number of steps n because of the Markov assumption. Usually, these results hold only in average. In 
particular, the assumption of a Gibbsian field in space-time, as studied in [4j, is more general than Markovianity. A 
generalization of our approach to this context is unlikely because it spoils the equivalence of paths that have the same 
transition record. 
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